The main aim of this paper is to provide an analysis of gradient descent (GD)algorithms with gradient errors that do not necessarily vanish, asymptotically.In particular, sufficient conditions are presented for both stability (almostsure boundedness of the iterates) and convergence of GD with bounded,(possibly) non-diminishing gradient errors. In addition to ensuring stability,such an algorithm is shown to converge to a small neighborhood of the minimumset, which depends on the gradient errors. It is worth noting that the mainresult of this paper can be used to show that GD with asymptotically vanishingerrors indeed converges to the minimum set. The results presented herein arenot only more general when compared to previous results, but our analysis of GDwith errors is new to the literature to the best of our knowledge. Our workextends the contributions of Mangasarian & Solodov, Bertsekas & Tsitsiklis andTadic & Doucet. Using our framework, a simple yet effective implementation ofGD using simultaneous perturbation stochastic approximations (SP SA), withconstant sensitivity parameters, is presented. Another important improvementover many previous results is that there are no `additional' restrictionsimposed on the step-sizes. In machine learning applications where step-sizesare related to learning rates, our assumptions, unlike those of other papers,do not affect these learning rates. Finally, we present experimental results tovalidate our theory.
展开▼
